Using a data-driven approach for the development and evaluation of phenotype algorithms for systemic lupus erythematosus

Background Systemic lupus erythematosus (SLE) is a chronic autoimmune disease of unknown origin. The objective of this research was to develop phenotype algorithms for SLE suitable for use in epidemiological studies using empirical evidence from observational databases. Methods We used a process for empirically determining and evaluating phenotype algorithms for health conditions to be analyzed in observational research. The process started with a literature search to discover prior algorithms used for SLE. We then used a set of Observational Health Data Sciences and Informatics (OHDSI) open-source tools to refine and validate the algorithms. These included tools to discover codes for SLE that may have been missed in prior studies and to determine possible low specificity and index date misclassification in algorithms for correction. Results We developed four algorithms using our process: two algorithms for prevalent SLE and two for incident SLE. The algorithms for both incident and prevalent cases are comprised of a more specific version and a more sensitive version. Each of the algorithms corrects for possible index date misclassification. After validation, we found the highest positive predictive value estimate for the prevalent, specific algorithm (89%). The highest sensitivity estimate was found for the sensitive, prevalent algorithm (77%). Conclusion We developed phenotype algorithms for SLE using a data-driven approach. The four final algorithms may be used directly in observational studies. The validation of these algorithms provides researchers an added measure of confidence that the algorithms are selecting subjects correctly and allows for the application of quantitative bias analysis.

Introduction Systemic lupus erythematosus (SLE) is a chronic autoimmune disease of unknown origin. Clinical manifestations include fatigue, arthropathy, and involvement of nearly all organ systems, particularly cardiac and renal [1][2][3][4]. A review by Stojan and Petri of research on multicountry incidence rate estimates found the incidence rate of SLE to be between 1-9 cases per 100,000 person-years (PY) [5].
The use of real-world evidence (RWE) from observational data, including administrative claims and electronic health records (EHR) datasets, is critical for studying the epidemiology and clinical manifestations of SLE. Developing and applying accurate phenotype algorithms is central for using observational data for analyses of health conditions. We performed a literature search and found 58 journal articles which included phenotype algorithms for SLE. A phenotype algorithm is the translation of the case definition of a health condition or phenotype into an executable algorithm based on clinical data elements in a database [6]. The algorithms used included those with 1 or more codes for SLE in a patient's record. Many included requirements for laboratory results from anti-nuclear antibody (ANA) tests. Others included prescriptions for anti-malarial drugs or oral corticosteroids. We found five studies validating SLE algorithms using clinical adjudication after 2010. Each study was performed on a single dataset. The research by Barnado et al provided the most recent example of the validation of algorithms for SLE. They evaluated a wide variety of algorithms using variations of the number of codes in a patient's record for SLE (1-4 codes), presence of drugs for SLE including anti-malarials, corticosteroids, and disease-modifying antirheumatic drugs (DMARD), and results from ANA tests. They performed their analyses in a single database, the Vanderbilt University Medical Center's Electronic Health Record system. The algorithms were evaluated in a group of 200 randomly selected patients whose medical charts were reviewed and adjudicated by clinicians to confirm or refute the presence of SLE. The best performing algorithm was one with � 3 SLE International Classification of Diseases, Ninth Revision (ICD-9) code 710.0 counts, ANA positive laboratory result, ever use of DMARDs, and ever corticosteroids use. They found a positive predictive value (PPV) of 91%. They also found a PPV of 95% when they excluded subjects with Dermatomyositis (ICD-9 code 710.3) and Systemic Sclerosis (ICD-9 code 710.1). In another algorithm with � 3 SLE ICD-9 code counts and ever antimalarial use, they found a PPV of 88% without excluding subjects with Dermatomyositis and Systemic Sclerosis and a PPV of 91% when excluding subjects with Dermatomyositis and Systemic Sclerosis. Limitations of this study included the use of data from a single health center and the calculation of sensitivity based on subjects with a mention of SLE in their record. Specificity was not measured in this study.
The objective of this research was to develop and evaluate phenotype algorithms for SLE suitable for use in epidemiological studies using empirical evidence from data in observational databases. The development process was performed on multiple databases from several countries to increase generalizability. The evaluation process included all performance characteristics, including sensitivity and specificity, and were assessed independent of the inclusion of SLE codes in each subject's health record and without utilizing medical chart review.

Materials and methods
We applied a rigorous process to develop the algorithms in this study. We used an extensive literature review (details and results are in S1 File), along with several tools within the Observational Health Data Sciences and Informatics (OHDSI) tool stack for empirical analysis. The goal of instituting this process was to enhance the science of phenotype algorithm development. The full details of the process are described in S2 File. We developed and evaluated four phenotype algorithms for SLE to be analyzed in observational research (Table 1): two incident algorithms requiring at least 365 days prior look-back and at least one SLE code ("Incident, 1X") or at least two codes with a code for SLE 31-365 days after the first code ("Incident, 2X"); two prevalent algorithms requiring at least one SLE code ("Prevalent, 1X") or at least two codes with a code for SLE 31-365 days after the first code ("Prevalent, 2X"). Requiring a 365 day look-back for the incident algorithm will impact the sensitivity of the algorithm especially in databases where subjects are observed for shorter periods of time. Each of the algorithms was corrected for possible cohort entry date (index date) misclassification. This was achieved by allowing for signs and symptoms for SLE to mark the start of the condition if these were followed by a code for SLE within 90 days of the signs or symptoms (Fig 1). We found 90 days to be optimal to achieve good coverage of prior signs, symptoms, and drugs. In the period 365 to 91 days prior to the first diagnosis code for SLE, the rates of these signs and symptoms were significantly lower than in the 90 to 0 days prior to index date. Thus the use of 90 days prior to index as the optimal time period choice. Signs and symptoms included codes for malaise, joint pain, and low back pain, consistent with the signs and symptoms of SLE [2,[7][8][9]. There were also prescriptions for drugs often used in the treatment of SLE and its symptoms, such as prednisone and methylprednisolone. The presence of these diagnosis and drug codes in the 90 days prior to the index date may indicate that the condition began prior to the appearance of the first SLE diagnosis code and the clinical diagnosis. From these results we inferred that there was likely index date misclassification and corrections to account for the prior signs, symptoms, and drugs were applied to improve the index date accuracy. The complete set of ICD, Read, and SNOMED codes used in the algorithms are in the S3 File.
We evaluated the algorithms against a network of nine observational databases. All databases were standardized to the OMOP Common Data Model, which provides the capability for the phenotype algorithms to be developed and consistently applied across the data sources. The nine databases were a mix of administrative insurance claims, electronic health records, and general practitioner databases from Germany, France, Australia, Japan, and US. The description and details for each of the databases are in Table 2. For each of the cohorts in each of the databases, we extracted patient-level data on patient demographics, diagnostic conditions, laboratory measurements, diagnostic and medical procedures, and drug exposures. The Optum and IBM MarketScan databases used in this study were reviewed by the New England Institutional Review Board (IRB) and were determined to be exempt from broad IRB approval, as this research project did not involve human subject research. Based on Ethical Guidelines for Epidemiological Research issued by the Japanese Ministry of Health, Labor and Welfare, ethics approval and informed consent for the JMDC database were not applicable for this study. The data from IQVIA were deemed commercial assets and there was no IRB applicable to the usage and dissemination of these result sets or required registration of the protocol with additional ethics oversight. We applied the OHDSI CohortDiagnostics tool (https://github.com/OHDSI/ CohortDiagnostics) to evaluate and compare algorithms at a population-level, characterizing overall count, incidence over time, index date breakdown, cohort overlap, and temporal characterization.
We applied the PheValuator [10] method to evaluate the performance characteristics of the algorithms against the databases. This method provides the complete set of performance

PLOS ONE
Phenotype algorithm development for systemic lupus erythematosus characteristics, i.e., sensitivity, specificity, and positive and negative predictive value. Using this method, we also evaluated algorithms from the Barnado et al study for comparison [11]. The two algorithms from Barnado et al were "Systemic Lupus 3X plus ever anti-malarial drugs" and "Systemic Lupus 3X plus ever anti-malarial drugs excluding dermatomyositis and systemic sclerosis". Using the semi-automated phenotype algorithm evaluation method Phe-Valuator, we eliminated the need for obtaining and reviewing subject's records. While algorithm validation results from chart review are considered the "gold standard", we have compared the results from PheValuator with prior studies using chart review and found excellent agreement between the two methods [12].
The demographic and clinical characteristics of those determined by the Incident, 2X algorithm were compared to those from a non-SLE randomly selected cohort of subjects. Non-SLE subjects were matched to the SLE cohort by age, sex, and year and month of SLE diagnosis in a ratio of 1 SLE:10 non-SLE subjects. The year and month of SLE diagnosis with the first clinical visit was matched in the same year and month in the randomly selected subject. We also required the same minimum look-back period, 365 days, in the matched cohort as was specified in the SLE cohort. Comparisons were made for characteristics observed in the year prior to SLE diagnosis (SLE cohort) or the matching visit date (non-SLE cohort). Standardized difference of the mean were calculated to assess difference between the cohorts [13].
The source code to implement the process used to develop and evaluate the algorithms is publicly available at (https://github.com/OHDSI/PhenotypeEvaluations/tree/main/SLE). This repository contains the literature search algorithm, the cohort definitions in JSON format, and the code to run CohortDiagnostics and PheValuator.

Results
We examined the characteristics of the subjects included by the algorithms. These characteristics may be viewed interactively at https://data.ohdsi.org/SLECohortDiagnostics. Subject counts in each of the databases ranged from about 117 subjects in France to about 180K in CCAE for the Incident, 1X algorithm ( Table 3). The counts were as expected based on the relative sizes of the databases indicating that all codes used were appropriate for each database. The counts were much higher in the US databases compared to the databases from outside the US. The reduction in the number of subjects in the Incident, 1X algorithm compared to the Incident, 2X algorithm was significant, ranging from about an 80% reduction in Germany and Australia to about a 50% reduction in Japan. This is graphically depicted in Fig 2 which shows the proportion of overlap between the algorithms in each database. Fig 2 also shows that the Incident, 2X algorithm is a proper subset of Incident, 1X algorithm, i.e., all subjects in the 2X algorithm were also in the 1X algorithm. We compared the diagnosed conditions, prescribed drugs, laboratory measurements, and clinical procedures of the subjects included in cohorts defined by the algorithms to see how  There were also differences, albeit fewer, in the Medicaid (MDCD) population, those generally of lower socioeconomic status. This dataset also showed differences in diagnosis codes for SLE (92% Incident, 2X v. 57% Incident, 1X, standard mean difference (SMD) 0.61) and prescriptions for hydroxychloroquine (27% Incident, 2X v. 15% Incident, 1X, SMD 0.21). The Medicare (MDCR) dataset, generally individuals 65 years and older, showed fewer significant differences in proportions between the two cohorts. However, there were also differences in diagnosis codes for SLE (83% Incident, 2X v. 49% Incident, 1X, standard mean difference (SMD) 0.54) and prescriptions for hydroxychloroquine (29% Incident, 2X v. 15% Incident, 1X, SMD 0.24). Overall, the relative proportions between the two cohorts for the majority of the characteristics in MDCR were closer to the 45˚line, indicating similar proportions between the cohorts compared to Commercial Claims and Encounters (CCAE) and MDCD. We examined the Incident, 2X algorithm for subject characteristics across the databases. A higher proportion of females compared to males with SLE were identified. The largest disproportionality was in MDCD where 91% of the subjects were female. Japan had the lowest disproportionality by sex with 68% of the subjects female. The type of clinical visit on first diagnosis of SLE was most commonly an outpatient or office visit. Less than 5% of the first diagnoses were made in an inpatient or emergency room visit with the exception being MDCD where

PLOS ONE
Phenotype algorithm development for systemic lupus erythematosus about 25% of the first diagnoses were made in an emergency room visit. This follows with other published studies examining the higher use of the emergency room in MDCD recipients [14]. Several differences were found in the demographic and clinical characteristics of the Incident, 2X algorithm compared to the matched non-SLE cohort. Differences in the standardized difference of the mean greater than 0.1 are considered imbalanced [15]. Imbalanced characteristics between the two cohorts included Black and White race (in MDCD and Optum EHR), diagnoses for rheumatoid arthritis, heart disease, and renal impairment. There were also differences in prescriptions for immunosuppressants and anti-thrombotic agents.
We also examined the index event, i.e., the diagnosis code that initiated the subject into the cohort. In the Incident, 2X algorithm, the most prevalent index event was a diagnosis code of "Systemic lupus erythematosus" (SNOMED code 257628; ICD-10CM and ICD-10GM M32.9 ("Systemic lupus erythematosus, unspecified"); ICD-9CM 710.0 ("Systemic lupus erythematosus")). There were a significant proportion of subjects whose index event was for a sign or symptom of SLE such as malaise, fatigue, anemia, or low back pain. These accounted for about 40-50% of the index events, indicating that there was likely a large amount of index date misclassification in subjects with SLE.
Incidence rates for SLE were similar from 2015 to 2020 across many of the databases. This may be partially attributed to the standardized use of ICD-10 coding starting in the US in 2015. In CCAE, MDCD, MDCR, Optum EHR, and Japan the rates were about 16 per 100,000 person years (PY). The rates in Germany were 1 per 100,000 PY. Rates in Australia and France varied considerably, likely due to the small sample size. The small number of SLE subjects in the Australian and French databases is likely due to the database being limited to general practitioners. In other databases which include specialists, such as rheumatologists, the sample size and incidence rates are higher and more stable. The incidence rates peaked in subjects in the 50-59 year old (YO) age group. One exception was in Japan where the incidence rates continued to increase with age through ages 70-79YO. Incidence rates in females were about 23-30 per 100,000 PY with the exception of Japan where the rate in females was about 18 per 100,000 PY. The incidence rates in males ranged from about 3-8 per 100,000 PY.
Evaluation of the performance characteristics of the four final algorithms was assessed using the PheValuator method (Table 4). Due to low subject counts, we were unable to calculate the performance characteristics for Australia and France. PheValuator requires a minimum of 200 subjects with a high likelihood of having SLE to produce an accurate model. This number was not satisfied in the Australian or French databases. As noted earlier, this may be due to the limitation of these databases to general practitioners. In general, the highest PPV estimates were for the two algorithms where a second diagnosis code for SLE was required 31-365 days after index. The highest sensitivity estimates were found in the two prevalent cohorts. The mean PPVs for the two algorithms where a second code was required was 87% (incident) and 88% (prevalent). This was reduced to 57% (incident) and 58% (prevalent) where only a single diagnosis code for SLE was required. The sensitivity for the two prevalent algorithms were 82% (single code required) and 55% (two codes required). This decreased to 39% (single code required) and 24% (two codes required) for the two incident cohorts. The highest mean F1 score was found in the prevalent, single code algorithms (65%) and the lowest was in the incident algorithms requiring two SLE diagnosis codes (37%). PPV estimates were higher in the US databases compared to Germany and Japan while the sensitivity estimates were similar between the databases. The estimates for PPV for the Barnado et al algorithms were 90% for both the algorithm using � 3 SLE ICD-9 code counts and ever antimalarial use, either with or without exclusion of subjects with Dermatomyositis and Systemic Sclerosis. This is similar to that found by Barnado et al (88% including subjects with Dermatomyositis and Systemic Sclerosis; 91% excluding subjects with Dermatomyositis and Systemic Sclerosis).

Discussion
In this study we used a data-driven approach to develop phenotype algorithms for SLE. The final four selected algorithms were developed to discriminate between incident and prevalent SLE populations, each with either higher sensitivity or higher specificity. In these algorithms, corrections were applied to address possible index date misclassification that might occur if the algorithms were limited to only diagnosis codes for SLE by adjusting the index date accounting for prior drugs, signs and symptoms. We validated each algorithm using the Phe-Valuator method to estimate the sensitivity, specificity, and positive and negative predictive values. To our knowledge, these are the first empirically derived phenotype algorithm definitions for SLE with the full set of validation performance characteristics. These performance characteristics may be used in studies requiring quantitative bias analysis (QBA). QBA provides quantitative estimates of the direction, magnitude, and uncertainty arising from systematic errors [16]. QBA, for example, may be used to estimate systematic errors in incidence rates, providing a more robust estimate. In QBA, the sensitivity and the specificity of the phenotype algorithm used within the study are included as parameters in the corrective equations. The results from the current study may be used for these corrections. The use of these algorithms with QBA in SLE observational research may improve the confidence in study results. Researchers may choose an algorithm for use in their observational studies based on fit for function according to the estimated performance characteristics. We found a large increase in PPV when a second code for SLE was added to the algorithm with a concomitant large decrease in sensitivity compared to the single code algorithm. These differences indicate that both the single code and the two code algorithms should be considered for use in studies depending on the study requirements. For studies requiring a more sensitive algorithm, the Prevalent, 1X algorithm should be used; for studies requiring a more specific algorithm, the Incident or Prevalent, 2X algorithm should be used.
There are several strengths to the present study. First, this study developed phenotypes using data from nine large datasets covering five countries and reflect subjects of a wide range of ages and from various socioeconomic backgrounds. Many previous studies examining the performance characteristics of algorithms for SLE used smaller datasets [17][18][19][20]. The data used for the development of the phenotypes was analyzed using multiple approaches, providing ancillary verification for each of our decisions in determining algorithm logic. The approach we used in this study uses publicly available, open-source software providing the capability for full result replication. Included in the supplemental information are the JSON files which provide fully reproducible phenotype algorithms. There were also several limitations to our study, which included the use of administrative datasets primarily maintained for insurance billing that is well-known to have significant deficits including coding inaccuracies [21]. In addition, the estimation of performance characteristics using the PheValuator methodology is dependent on the quality of the data in the dataset, which can vary substantially [22]. Incomplete signs and symptoms documentation in data could affect the accuracy of the index date. This may reflect differences between insurance claims databases, e.g., CCAE and EHR databases, e.g, Optum EHR. The generalizability of findings to uninsured populations may differ from the insured population observed in this study. Prescription drug treatments in claims and EHR data are not specifically associated with an indication which could affect the signs and symptoms and correction for index date misclassification and associated metrics. The algorithm validation was performed using a method involving predictive modeling of SLE rather than case review. This method does have the advantage of providing performance characteristics for multiple databases. It also provides the full set of performance metrics including sensitivity and specificity which are rarely provides in validation studies using case review [10]. Using this method we found similar results to a prior validation of SLE. The results from PheValuator have also been compared to the results from previously published validation studies and have demonstrated excellent agreement [12]. In the algorithm where an incident cohort definition was defined with only one diagnostic code it was not possible to determine if any of these were rule out diagnoses. Lastly, our study observed data from those subjects who presented for medical attention; those who did not seek medical attention but had the disease were not included and may affect the metrics in this study. Those with less severe disease may also have not sought medical attention.

Conclusions
This study developed and thoroughly evaluated phenotype algorithms for SLE based on a data-driven approach. The results of this effort yielded four final algorithms, which may be applied by other researchers to observational studies of SLE. The four algorithms include options for prevalent vs. incident cohorts as well as sensitive vs. specific definitions. The validation metrics provided with these algorithms increases the confidence in these algorithms to identify SLE subjects and also enables the application of quantitative bias analysis.